The statements in this section merely provide background information related to the present disclosure and may not constitute prior art. The Boeing owned patent applications for U.S. Ser. Nos. 10/817,476 and U.S. Ser. No. 11/382,523, teach systems and methods to automatically register one image to a second image, where the two images are typically obtained by different means at different times and under different conditions. The first image, which may be identified as an image from a sensor, conveys a view of a scene from a particular vantage point. The second image, which may be identified as a reference image, conveys a view of the same scene from a typically different vantage point, and will typically have a different appearance from the sensor image, due to differences in vantage point and image collection conditions, such as different sensor modalities (e.g. SAR sensor and optical reference image). This reference image may come from a reference image database covering at least the scene imaged by the sensor, but typically comprises a much larger area of which the scene is only a small part. Scene content depicted in the sensor and reference images will also typically differ, as the sensor image may contain scenic features which were not present when the reference image was obtained. For example, a mobile target may be present in the sensor image, which is not shown in the reference image. Alternatively, features in the scene such as roads or houses or trees may have changed since the reference image was initially obtained, with new roads or houses being built, or trees growing to be large or having been cut down.
Registration of the two images can be accomplished by the systems and methods taught in the afore-mentioned Boeing patent applications, as long as the scenic or feature content between the sensor and reference images share enough information in common. Such a condition typically occurs in most situations.
With the foregoing Boeing patent applications, the reference image is accompanied by a method to associate individual pixels in the reference image to specific three dimensional locations in the scene depicted. For a single reference image accompanied by a digital elevation model (DEM—typically a matrix or array of elevation measurements covering the reference image area), accuracy of the scene locations so derived is typically limited primarily by the accuracy of the elevation model. The accuracy of the registration may also be limited by the accuracy of the method used to associate the other two spatial coordinates with locations in the reference image. In the case when the single reference image is one image from a Digital Point Position Data Base (DPPDB), a product of the National Geospatial-Intelligence Agency, and the elevation model is a Digital Terrain Elevation Dataset (DTED—a DEM in a specific file format), from the same Agency, the accuracy is limited to what is termed the “mono-intersection” method accuracy of the DPPDB. In the “mono-intersection” method, a point measurement for a scene is obtained by calculating a pixel location in the DPPDB image using a three dimensional scene coordinate, and then adjusting the three dimensional scene coordinate until the calculated location has the same column and row location as the desired pixel of interest.
In this limited method, the reference image or image pair can be any images having an accurately predefined relation to a local coordinate system of the scene depicted in the image or images. The elevation model can likewise be any source of three dimensional coordinate data that describes the scene surface's shape, including a computer-aided design model.
The three dimensional coordinate is obtained by using a latitude, longitude coordinate pair, with an elevation approximated using the DEM measurements near that specific location. The elevation approximation is obtained by interpolation of height measurements given in the DEM from locations nearby the specified latitude, longitude location. The accuracy is limited, however, because each given pixel location in the two dimensional image can be derived from a multiplicity of points in a three dimensional volume of three dimensional coordinate points, as the three dimensions (here, latitude, longitude and elevation, but potentially any three x, y, z coordinates three dimensional position recording) are mapped redundantly into two image dimensions (column and row).
If the elevation model were exact, so that at each specific latitude and longitude location the model gave the correct elevation for that location, then this “mono-intersection” method could give best accuracy. Most desirable would be a model that contained an accurate elevation measurement for each pixel in the reference image. It is possible to obtain elevation models that are more accurate and have greater spatial resolution than those provided by DTED, or more accurate than other readily available elevation models from other providers, such as the United States Geophysical Service. However, in practice, exact elevation models, or even high accuracy models having high spatial resolution of elevation measurements to agree with the spatial resolution of the reference image, can not be obtained easily or be used easily in a practical situation requiring rapid measurements. Such alternative, exact elevation models also typically require large amounts of data storage capability, which can be onerous for smaller systems, for example on-board systems carried by various forms of airborne and land based mobile platforms.
In some cases, the reference image may consist of a stereo pair of images, whereby both images depict the same scene area, but from slightly different perspective vantage points. The slightly different perspectives of the two stereo reference images allows stereographic processing to determine highly accurate three dimensional location coordinates for points in the scene, when depicted in both reference images. Such stereo measurements give the highest accuracy for measurements of points in the two images. In standard stereo measurement, the points in the two images must both be visible to the operator, else measurement accuracy may be reduced.
In general, image registration may not achieve sufficient accuracy, because it is limited by the accuracy of the registration, and by the accuracy of the 3-D scene coordinate associations for reference image and elevation model. Under existing art, even when using two stereo reference images, image registration may not achieve sufficient accuracy. Once the sensor image is registered to a first reference image, pixel locations in the sensor image are uniquely associated with pixel locations in the reference image. As taught in the two previously cited Boeing patent applications, this allows a point of interest in the sensor image, such as target seen only in the sensor image, to be associated with a point in the reference image where no target is seen. Thus, using the scene 3-D coordinate associations of the reference image, this point in the sensor image is associated with a 3-D scene coordinate. The accuracy of the registration governs the accuracy of locating this unseen point in the reference image, and the accuracy of the 3-D scene coordinate associations with the reference image governs the accuracy of the 3-D scene coordinate assigned to this unseen point. By means described in the two Boeing patent applications, and as discussed above, the location in scene coordinates of this point of interest in the sensor image, and its corresponding unseen point within the reference image, can then be determined. The problem, then, is that the accuracy of that measured 3-D scene location corresponding to the point in the sensor image and the unseen point in the reference image may be less than needed to accomplish the measurement goal.
With stereo reference images, registration of the sensor image to one or both of the stereo reference images is necessary also, so that 3-D scene locations can be derived for points in the sensor image that are not visible one or both of the stereo reference images. According to existing art, stereo measurement is most accurate when the two stereo images are measured simultaneously, and the point to be measured is visible in both stereo reference images. However, registering the sensor image to the two stereo reference images must be done separately. This will result in two different 3-D scene locations being associated with the point of interest in the sensor image, each of which separately has an error associated with registration, and an error of the 3-D scene coordinate association with the corresponding stereo reference image, and, neither of which has the accuracy of a most accurate stereo measurement. If the point of interest is visible in both stereo reference images, most accurate stereo measurement is possible, but may be difficult to perform in many situations, such as when stereo viewing is unavailable or operational timelines are short, as is the case in tactical fighter operations, real-time robot guidance, and other applications of interest. This invention operates to overcome that deficiency.
Another problem is validating the measurement method for proper error estimation of measurements. Such validation is required by, and performed by, the National Geospatial-intelligence Agency, and is conducted to ensure that target point measurement error is properly predicted. An image registration process by itself introduces elements into the error propagation that are difficult to predict, and usually contain factors involving probability that may not be acceptable under NGA validation rules. For example, a probability of correct match could be one of those factors.
Thus, there exists need for a method and system that even more accurately enables a selected point within a scene of interest to be registered to a reference image, and then the reference image used to enable three dimensional coordinate information of the selected point to be determined in a way that ensures that optimum accuracy of the location of the selected point will be obtained.